Hello,
we are using prometheus to trigger alerts in pagerDuty. But there is also a prod stage with an onCall Dev. If we test the prod stage, this will interrupt and confuse this Dev. Is there a good way to test also the prod stage by using the prometheus alert manager? Maybe using the info severity or something like that?
Greetings.
Hi @Frost208,
There are a few things you could do here. If you have a schedule in your escalation policy, you could create an override so that you or a different user is on call for the service before doing the test.
You could also configure this service to trigger low urgency incidents only. This way, instead of calling the on-call responder, it would follow the low urgency notification rules that the user has configured (which is by default just an email).
If your service is configured to use alert severity, then you could send in your test events with "severity": "info", as you suggested.